DOCUMENT RESUME 



ED 403 230 



SP 037 088 



TITLE 



INSTITUTION 
SPONS AGENCY 

PUB DATE 
CONTRACT 
NOTE 

AVAILABLE FROM 
PUB TYPE 



Questioning and Understanding To Improve Learning and 
Thinking (QUILT): The Evaluation Results. A Proposal 
to the National Diffusion Network (NDN) , Documenting 
the Effectiveness of the QUILT Professional 
Development Program. 

Appalachia Educational Lab., Charleston, W. Va. 

Office of Educational Research and Improvement (ED) , 
Washington, DC. 

Feb 94 
RP91002002 
29p . 

Appalachia Educational Laboratory, Inc., P.0. Box 
1348, Charleston, WV 25325-1348; (800) 624-9120. 
Reports - Evaluative/Feasibility (142) 



EDRS PRICE 
DESCRIPTORS 



IDENTIFIERS 



MF01/PC02 Plus Postage. 

Classroom Techniques; Cognitive Style; Elementary 
School Teachers; Elementary Secondary Education; 
Faculty Development; Inservice Teacher Education; 
^Learning Processes; ,v Ques t i oning Techniques; 
Secondary School Teachers; ^Student Behavior; 
^Teacher Behavior; Teacher Improvement; Teaching 
Methods; *Th inking Skills 
*QUILT (Staff Development Program) 



ABSTRACT 

The goals of the QUILT program are to increase and 
sustain teacher use of classroom questioning techniques and 
procedures that produce higher levels of student learning and 
thinking, and to increase the incidence of student responses at 
higher levels of cognition. Educational research has established 
relationships between discrete questioning behaviors and student 
learning and thinking outcomes. Research also documents that most 
teachers have not incorporated effective questioning behaviors into 
their classroom repertoire. QUILT is intended to help teachers align 
their classroom questioning practices with "best practices” by 
supporting their change efforts over time. Local teams participate in 
a t raining-f or~t rainers that provides the knowledge, skills, and 
materials required to implement QUILT. Activities for faculty 
include: workshops; collegiums; observation sessions; and specified 
classroom applications. After one year’s participation in QUILT, 
teachers in 13 schools showed significant gains in knowledge, 
understanding, and application of concepts related to effective 
questioning. A random sample of these teachers showed positive 
changes in their questioning techniques. Also, student answers to 
teacher questions were more often at cognitive levels above simple 
recall following teachers’ participation in QUILT. Attachments 
include: the QUILT model; an annotated bibliography; a list of 
participants in QUILT field tests; and summaries of workshop 
evaluations and observation data. (ND) 
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Questioning and Understanding to Improve Learning and Thinking 

(QUILT) 

Abstract 



Goals: The primary goal of the program is to increase and sustain teacher use of classroom 
questioning techniques and procedures that produce higher levels of student learning and 
thinking. A related goal is to increase the incidence of student responses at higher levels of 
cognition. 

Purposes and Needs Addressed: Educational research has established relationships 
between discrete questioning behaviors and student learning and thinking outcomes. Research 
also documents that most teachers have not incorporated effective questioning behaviors into 
their classroom repertoire and are not likely to do so without a structured, sustained, long-term 
effort to change routinized questioning behaviors. QUILT is intended to help teachers align 
their classroom questioning practices with "best practice" by supporting their change efforts 
over time. 

Method of Operation: Local facilitation teams participate in a training-for-trainers that 
provides the knowledge, skills, and materials required to implement QUILT. Ideally, the entire 
faculty of an adopting school actively participates in the QUILT program that consists of: a 
three-day Induction training that presents the knowledge bases; seven 90-minute collegiums 
scheduled across the school year; seven opportunities to work with a partner to observe and be 
observed; and specified classroom applications. 

Audience: All teachers, K-12, are the intended audience for the program. 

Claims: After one year's participation in QUILT, teachers in 13 schools showed significant 
gains in knowledge, understanding, and application of selected concepts related to effective 
questioning. A random sample of these teachers showed significant, positive changes in the 
following behaviors: decrease in number of teacher questions, use of wait times I and II, 
redirection of questions, percent of questions at higher cognitive levels, designation of respon- 
dents before questions, and decrease in repetition of student answers. Student answers to 
teacher questions were at cognitive levels above simple recall significantly more often follow- 
ing teachers' participation in QUILT. 



Basic Information 

A. Project Title: Questioning and Understanding to Im- 
prove Learning and Thinking (QUILT) 

Location: Appalachia Educational Laboratory (AEL) 
Contact Person: Sandra Orletsky, Project Director, P. O. 
Box 1348, Charleston, WV 25325; 304/347-0400 

B. Original Developer 

The QUILT program was developed at AEL as part of the 
work of the School Governance and Administration (SGA) 
program, directed by Sandra R. Orletsky. The major authors 
and codevelopers are Jackie A. Walsh, educational consultant, 
and Beth D. Sattes, R & D specialist at AEL. The evaluator for 
the QUILT program is J. Jackson Barnette, professor at the 
University of Alabama. 

The Appalachia Educational Laboratory (AEL), Inc., is a 
nonprofit corporation that works with educators in ongoing R 
& D-based efforts to improve education and educational op- 
portunity. AEL serves as the Regional Educational Laboratory 
for the states of Kentucky, Tennessee, Virginia, and West Vir- 
ginia. AEL works to improve professional quality, curriculum 
and instruction, community support, and opportunity for ac- 
cess to quality education by all children. 

C. Years of Project 

Developed: May 1989 - June 1991 
Operated: June 1991 - present 
Evaluated: June 1991 - June 1992 constituted the 
major field test 

Disseminated: June 1992 - present 

D. Source of Development and Dissemination Funding 



FY 89 - FY 90 


$140,000 


from the U. S. Department 
of Education, contract 
400-86-0001 


FY 91 


230,000 


from the U. S. Department 
of Education, RP91002002 


FY 92 


181,000 


from the U. S. Department 
of Education, RP91002002 


FY 93 


176,000 


from the U. S. Department 
of Education, RP91002002 


Listed above are the funds (from the Regional Educational 
Laboratory contracts) expended on the development, evalua- 
tion, and dissemination of the QUILT program. Local schools, 
local school districts, and states within the AEL Region have 



also contributed to development and evaluation costs; how- 
ever, it is impossible to attribute precise dollar figures to these 
contributions. It is important to note that without the active 
participation and financial backing of these local schools, the 
development and evaluation could not have been done to such 
high and rigorous standards. 

Description of Program 

A. Goals 

The primary goal of the Questioning and Understanding 
O mprove Learning and Thinking (QUILT) program is to 
ease and sustain teacher use of classroom questioning 



techniques and procedures.that produce higher, levels of stu^ 
dent learning and thinking. 

B. Purposes and Needs Addressed 

Gassroom questioning is perhaps the most used — and the 
most misused — of all instructional processes. On the average, 
teachers dedicate approximately 40 percent of classroom in- 
structional time to the asking and answering of questions 
(Doyle, 1986); in machine-gun fashion, they pose an average of 
40-50 questions in a typical 50-minute class segment. How- 
ever, most of these questions are not well prepared and do not 
serve the purpose of prompting students to think (Dillon, 
1988). QUILT attempts to help teachers improve the quality of 
the questions they pose and, in the process, reduce the number 
of questions asked. This results in a more reflective classroom 
environment — where teachers and students alike have time to 
think about the content and issues under study. 

A tremendous gap exists between "what is" and "what 
should be" with regard to teacher questioning behaviors. In 
spite of the plethora of articles, books, and workshops on the 
topic of classroom questioning, teacher behavior remains rela- 
tively unchanged over 100 years of classroom observations 
and research (McNamara, 1981). From the inception of their 
work, QUILT developers sensed that the real need was to 
design a program that would support teachers in personal 
behavior changes — one that would help them change or alter 
firmly entrenched patterns. 

C. Intended Audience 

QUILT is a program designed for all teachers, K-12, in all 
content areas. 

D. Background, Foundation, and Theoretical Framework 

Development. In 1987, the Appalachia Educational Labo- 
ratory (AEL) contracted with Dr. Jackie A. Walsh to develop a 
12-hour workshop on effective classroom questioning. The 
"Effective Questioning Workshop: Good Questions Don't Just 
Happen" was frequently requested in AEL's four-state Re- 
gion; evaluations at the end of each training were positive and 
affirming of both (a) the importance of the content and (b) the 
appropriateness of the activities incorporated into the design 
of the workshop. Although the workshop was popular with 
educators in the Region, AEL staff questioned the wisdom of 
spending limited resources to conduct workshops. It seemed 
to be well accepted in the literature that workshops resulted (at 
best) in increased awareness but rarely in meaningful or mea- 
surable changes in teacher performance. 

During this same time, the Kentucky Association of School 
Administrators (KASA) requested help for school administra- 
tors responsible for planning and delivering "effective" staff 
development. One of their criteria for "effective" was that it 
incorporate research about the change process and be a long- 
term program (as opposed to the predominantly one-shot 
programs that proliferate in schools). 

The two agencies agreed to work collaboratively on effec- 
tive staff development, and thus was bom the KASA-AEL 
Study Group on Effective Questioning. Membership was 
comprised of 20 teachers and administrators from five Ken- 
tucky local school districts, KASA staff, and AEL staff. The 
study group worked for a full year Qune 1989-1990) in the 
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development and pilot testing of concepts and materials that 
would become the QUILT program, which was not only firmly 
grounded in research and theory but also was well connected 
to the reality of living; breathing teachers and administrators in 
public schools. During the 1990-91 school year, AEL staff 
piloted the program in a West Virginia school district. By June 
1991, revisions had been made and QUILT was ready for its 
first large-scale implementation. 

Questioning framework. Questioning is widely accepted 
as the centerpiece of the teaching-learning process. Question- 
ing is a powerful tool for teachers at all grade levels and in all 
disciplines. Good questions, effectively delivered, facilitate 
student learning as they serve to motivate and focus student 
attention, provide opportunities for practice and rehearsal, 
and function as yardsticks of how well students are mastering 
content. Further, questions stimulate students to process infor- 
mation at deeper levels. Questions hold the key to improving 
student learning and thinking. The QUILT program presents 
the questioning-answering-reacting processes as dynamic and 
interrelated systems whose parts interact to affect the out- 
comes of the teaching-learning transaction. (See the complete 
QUILT Model in Attachment #1.) 

Research has informed us about effective questioning prac- 
tice and, yet, as described in the bulleted items below, research- 
ers have also documented that standard practice falls far short 
of the ideal. 

• Wait Time I. Teachers require students to respond almost 
instantaneously to questions, allowing less than one sec- 
ond for students to think through their answers. How- 
ever, in classrooms where teachers wait three to five sec- 
onds, students give longer responses, answer more fre- 
quently at higher levels of cognition, demonstrate more 
confidence in their answers, and ask more questions 
(McGlathery, 1978; Rowe, 1986). 

• Wait Time II. Teachers react immediately to a student 
response, waiting an immeasurably short amount of time 
before providing feedback or making another instruc- 
tional move. In classrooms where teachers wait three to 
five seconds after the initial student response, students 
answer more completely and more correctly; exhibit more 
speculative and inferential thinking; ask more questions; 
increase interactions with other students; and demon- 
strate more confidence in their responses (Rowe, 1974; 
Garigliano, 1972; Gooding et al„ 1983). 

• Asking questions at all cognitive levels. About 75 to 80 
percent of the questions posed in both elementary and 
secondary classrooms are at the knowledge or recall level 
(Dillon, 1988; Gall, 1984; Haynes, 1935; McGlathery, 1978). 
However, when students are afforded opportunities to 
answer questions at higher cognitive levels, they demon- 
strate an ability to analyze, synthesize, and evaluate; they 
also score better on tests measuring recall and understand- 
ing of that content (Red field & Rousseau, 1981). 

• Redirecting questions. Teachers typically answer ques- 
tions when students do not answer or do not give the 
answer the teacher was seeking. However, when teachers 
pose one question to multiple students, students are held 
more accountable for answering all questions; addition- 



ally, the interaction among and between students increases 
(Omstein, 1988; Riley, 1981). 

• Designating a respondent. Teachers frequently name a 
student to answer a question before posing the question. 
However, when teachers name a student after posing a 
question, all students are more likely to attend to the 
question and prepare a covert response (Gall, 1984; 
Omstein, 1988). • 

• Repeating student answers. Teachers typically repeat 
student answers; however, when teachers do not repeat 
answers, students pay greater attention to and show in- 
creased respect for their classmates* responses (Omstein, 
1988). 

Staff development framework. QUILT developers con- 
sidered the design of the staff development process to be as 
critical to program success as that of content selection and 
sequencing. Hence, as they designed the program structure, 
they drew from the knowledge bases on change theory, orga- 
nizational culture, effective staff development, andragogy,and 
related fields. QUILT recognizes that change begins with 
individuals, not organizations, and that changing one's behav- 
ior is difficult; embedded in the QUILT program are support 
structures to assist with this change. 

• Program activities are phased over an entire school year 
acknowledging that change is a process — not an event — 
that occurs over time (Hord et al., 1987). Local QUTLT 
facilitators receive training in the change process includ- 
ing the Concems-Based Adoption Model (CBAM) so that 
they can better serve in the helping role to their colleagues. 

• The QUILT program is intentional in its effort to construct 
a culture that promotes and sustains change. QUILT 
encourages the development of a shared vision, the use of 
symbols and metaphors, the development of a common 
vocabulary, celebration of successes, and other activities 
associated with culture-building (Deal and Kennedy, 1982). 

• QUILT is consistent with staff development literature that 
maintains that teachers learn and improve performance 
when provided opportunities to: (1) acquire a knowledge 
base, (2) observe demonstrations, (3) practice new behav- 
iors, and (4) receive feedback on their own performance in 
the classroom (Joyce and Showers, 1982). These four 
components are hallmarks of the QUILT program. 

• All components of the QUILT program were crafted by 
reference to well established principles of adult learning. 
Two principal hallmarks are interaction with peers and 
individual reflection, considered to be the most important 
ingredients for successful adult growth and development 
(Levine, 1989). A major component of QUILT is self- 
assessment and personal goal setting. The design of the 
program incorporates a number of support structures for 
individuals seeking to change — not the least of which is a 
strong collegial association. 

E. Features: How the Program Operates 

Scope. QUILT offers the opportunity for an entire faculty 
to focus their growth and development for a full year upon a 
process central to all instruction: effective classroom question- 
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ing. While QUILT is complementary to all other staff develop- 
ment initiatives, it is designed to satisfy a school's inservice 
requirements for a year. 

Curriculum and instructional approach. The content of 
the QUILT program is embodied in the QUILT Model (Attach- 
ment #1). Teachers participate in the following four instruc- 
tional components: 

• Induction training. This intensive, three-day (18-hour) 
training usually occurs as a preschool (August) session. 
Embedded in the induction experience are a presentation 
of knowledge and theory, demonstration of behaviors and 
skills, and many opportunities to apply and practice. 

• Collegiums. Teachers attend seven 90-minute seminars 
over the course of a school year (see box below). These 
sessions, held every four to five weeks, provide opportu- 
nities to (1) share successes and problems related to the use 
of discrete questioning behaviors, (2) review specific con- 
tent, (3) practice /apply associated skills and behaviors, (4) 
plan for classroom use, and (5) plan for work with a 
partner. Each collegium has a companion practicum com- 
prised of partnering and individual classroom use. 

• Partnering- At the beginning of the school year, each 
QUILT teacher is matched with a partner with whom to 
work as classroom implementation proceeds. Partners 
observe one another — using QUILT observation forms — 
and provide feedback. The observed partners demon- 
strate specified QUILT behaviors; hence, partners serve as 
"coaches/' 

• Classroom implementation. Teachers work indepen- 
dently to implement suggested strategies and, very im- 
portantly, to teach their students effective questioning 
processes. 



Learner activities. The activities — which include a bal- 
ance of interaction with peers and personal reflection — are 
targeted toward attainment of the purpose and objectives that 
are clearly stipulated for each segment. The seven collegium/ 
practicum segments comprise the substance of the QUILT 
program. The objectives for each of these are shown in the box 
below. 

Learning materials. Teacher materials for the complete 
QUILT program include: QUILT Induction Manual , seven indi- 
vidual teacher booklets (one for each collegium), multiple 
copies of five observation forms for partners' observations, 
Teaching and the Art ofQuestioningby]. T. Dillon (PDKFastback 
194), and set of seven teacher "cue cards" to assist with class- 
room implementation. 

QUILT is a turnkey training program wherein each adopt- 
ing school sends a training team to participate in a six-day 
training-for-trainers. The following materials support the ef- 
forts of the training team: QUILT Induction Lecturettes, QUILT 
Collegiums and Practicums: A Facilitator's Guide , QUILT Read- 
ings, eight videotapes for trainers, 149 transparencies (for use 
in induction and collegiums), camera-ready copy for QUILT 
poster quotes, and agendas and other handouts to support 
local implementation. 

Staff activities and staffing patterns. The preferred model 
is for a school to designate three to five individuals to serve as 
a training team. This team should be comprised of teachers 
and at least one administrator. Time for teachers to attend the 
induction training and to participate in collegiums and 
partnering activities will need to be integrated into the school 
calendar. A coordinator, to assist with organization, schedul- 
ing, and resource management, is also needed. 

Staff development activities. The designated training 
team attends a six-day training-for-trainers event that prepares 
them to deliver the three-day induction training, organize their 



Collegium/Practicum Objectives 



One: Prepare to QUILT— Objectives: (1) To renew my per- 
sonal commitment to change through QUILT, (2) To decide what 
my students need to know about QUILT, (3) To emphasize to my 
students the importance of Wait Time I and Wait Time II. 

Two: Prepare to Partner — Objectives: (1) To adopt the QUILT 
approach to partnering, (2) To observe and be observed using 
"Wait Times I and U^QUILT Observation Form", (3) To give and 
receive feedback; (4) To improve my use of wait times. 

Three: Improve Response Patterns— Objectives: (1) To use 
my partner's feedback to enhance questioning and responding in 
my classroom, (2) To increase the number of response formats 
used in my classroom, (3) To eliminate any action zones that may 
be operating in my classroom, (4) To try out the ' Round-robin 
Questioning" technique in my classes. 

Four: Create the Question — Objectives: (1) To formulate 
questions with consideration of purpose and focus, (2) To formu- 
late questions that engage student thinking at a variety of cogni- 
tive levels, (3) To work with my partner to analyze and classify my 
questions — using "Prepare the Question — QUILT Observation 



Five: Polish the Question — Objectives: (1) To phrase my 
questions so that my students are clear about what I am asking, (2) 
To write pivotal questions before asking them in class, (3) To work 
with my partner to revise and refine my written questions, (4) To 
teach my students how to question and respond at all cognitive 
levels. 

Six: Questioning for Discussion— Objectives: (1) To be pur- 
poseful in the use of questions for recitation and discussion, (2) To 
know how to plan for and sustain class discussion, (3) To know 
and use alternatives to the asking of questions during class discus- 
sions, (4) To teach students about the differing purposes of recita- 
tions and discussions, (5) To observe and be observed using 
"Teacher Reactions During Discussion — QUILT Observation 
Form." 

Seven: QUILTing— Objectives: (1) To assess my use of QUILT 
behaviors, (2) To reflect upon the benefits of the QUILT program 
for improving learning and thinking in my classes, (3) To commit 
to continued use of QUILT in my classes, (4) To observe and be 
observed using "Process Student Responses — QUILT Observa- 
tion Form." 



school for implementation, and facilitate the first two 
collegiums. A one and one-half day "booster" training-for- 
trainers session brings the training teams back together for 
additional facilitation training (including change facilitation) 
and training in delivery of collegiums 3-7. These training-for- 
trainers events are cofacilitated by Walsh and Sattes, 
codevelopers of QUILT. Local trainers receive training in 
generic facilitation and presentation skills. The majority of 
teachers comprising past training teams had little to no prior 
experience presenting workshops. Maximum number of par- 
ticipants for training-for-trainers events: 50. 

Management activities. A local coordinator should be 
designated to: (1) schedule induction and collegiums; (2) 

arrange for stipends to school trainers (and to teachers, if 
possible); (3) arrange college course credit and/or inservice 
credit; (4) facilitate the matching of partners; (5) support 
partnering activities (e.g., obtain substitute teachers so that 
partners can observe and engage in feedback conferences six 
times during the year); (6) handle arrangements for training 
sessions including space, refreshments, supplies, etc.; and (7) 
communicate with AEL QUILT project staff. 

Monitoring and evaluation procedures. QUILT requires 
that adopting schools collect and forward participant evalua- 
tions of induction training and of each collegium. The most 
significant type of project assessment is completed by the 
teachers themselves as they engage in cross-observations with 
their partners using QUILT observation instruments. 

F. Significance of Program Design as Compared to Similar 

Programs 

An extensive review of literature and dialogue with col- 
leagues across the nation uncovered a plethora of programs 
designed to improve classroom questioning. (See Attachment 
#2 for annotated listing.) Most of these were one-shot work- 
shops (one hour to three days in length) that had no provision 
for followup or evaluation of results. None of them go beyond 
presentation of research-based content to the design of a pro- 
gram structure that would assist teachers in changing over 
time. QUILT developers concluded that while the literature 
review provides testimony to the widespread concern and 
interest in classroom questioning, it also confirms that there is 



no long-term program that has.amassed evaluation evidence 
to demonstrate that it makes a difference in teacher perfor- 
mance in this area. QUILT appears to be unique in both these 
regards. 



Potential for Replication 

A. Settings and Participants (Development and Evaluation 

Sites) 

Nearly 400 teachers from 13 schools in 13 school districts 
participated in the 1991-92 field test of the QUILT program. 
Included were 163 teachers in three large, comprehensive high 
schools (two in Kentucky and one in Virginia); 34 teachers in a 
junior high school (in West Virginia); and 159 teachers in eight 
elementary schools (three in Tennessee, one in Virginia, and 
four in West Virginia). Additionally, 41 secondary teachers 
participated from Fort Knox Community Schools [a section VI, 
U. S. Department of Defense (DOD) unit]. As a total group, 
these teachers had an average of 14 years of teaching experi- 
ence; 75 percent were female; 95 percent were Caucasian; 42 
percent were 40-49 years of age; and 40 percent had a bachelor' s 
as their highest degree. The lack of ethnic diversity in the 
teachers mirrors the population of their communities and the 
Appalachian portion of the four-state Region served by AEL. 

Teachers at all grade levels (K-12) and 20 different second- 
ary content /subject areas implemented QUILT over the 1991- 
92 school year. Ten percent (10%) of the total were in special/ 
exceptional education. While approximately two-thirds of the 
secondary teachers instructed in the core academic subjects, 
other participants taught electives including computer courses, 
foreign languages, art, physical education, and the vocational- 
technical fields. At least two schools successfully included 
teacher aides in their implementations. The total included 11 
public schools from 11 different school districts; one parochial 
school, from the Diocese of Wheeling-Charleston, West Vir- 
ginia; and the DOD school in Fort Knox. The public and 
parochial schools reflect the population of the Appalachian 
Region — predominantly rural, Caucasian, working class in 
nature; the DOD school has greater racial and cultural diver- 
sity. 

Vignettes (shown in the box below) of three participating 
schools characterize the nature and diversity of the Region and 
the client groups served. 



Rustburg High School, Campbell 
County, Virginia: This school serves 
850 students in a stable, blue-collar com- 
munity of 50,000 residents. The high 
school diploma is the highest degree at- 
tained by 64 percent of the parents of 
Rustburg; 22 percent of the parents did 
not complete high school. Approxi- 
mately 26 percent of these students go on 
to college after spending all four years 
(grades 9-12) at this high school; the drop- 
out rate is 7 percent. About 16 percent of 
the students are African-American. 



East Bank Junior High School, Kanawha 
County, West Virginia: 580 students 
(grades 7-9) attend this school where the 
attendance rate was 87 percent and the drop- 
out rate 13 percent during the 1991-92 school 
year. Ninety-five percent (95%) of the stu- 
dents are Caucasian; 50 percent receive free 
and reduced lunches; 50 percent come from 
one-parent homes. Only 50 percent of the 
students have telephones in their homes, 
one clue to the relatively low socioeconomic 
status of the majority of students served by 
this school. 



Bel Aire Elementary School, 
Tullahoma, Tennessee: 375 elemen- 
tary students (grades K-6) attend this 
school operated by the small, indepen- 
dent city district located in this small 
town (17,000 population). Ninety-five 
percent (95%) of the town's popula- 
tion came from elsewhere in the nation 
to this community, whose students 
have a high level of education. The per 
capita income of the county is $14,028; 
32 percent of the Bel Aire students 
qualify for free /reduced lunches. 
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QUILT was developed for the total-universe of teachers— 
serving students of all ability levels and backgrounds in all 
grades and subject areas. While the teachers and students who 
participated in the field test were neither urban nor racially 
diverse, this does not mean that the program will not serve the 
needs of urban educators equally well. The majority of the 
research on effective questioning that informed development 
of the QUILT model was conducted in urban settings, often 
with at-risk students. Additionally, QUILT developers were 
sensitive to the broader issues of diversity in materials and 
workshop designs. 

B. Replicable Components and Documentation 

All elements of QUILT are transportable. The 13 schools in 
the field test successfully implemented QUILT during 1991-92, 
as did 21 school teams in the succeeding two years (1992-93 
and 1993-94). 

C. User Requirements and Costs 

The costs and requirements for implementing the QUILT 
program are outlined in Table 1 on the next page. The first 
column identifies a timeline and costs for each adopting school. 
Additional costs to the school or district are listed in the final 
column, "Optional Incentives." No price is attached to these 
optional incentives because costs would vary from district to 
district; many districts are successful in having some of these 
services donated by local community businesses. None of 
these "optional incentives" is required to implement QUILT; 
however, each of them contributes to the success of the pro- 
gram, we believe, by giving recognition and rewards to the 
trainers and teachers who participate. 

The second column describes timeline, responsibilities, 
and costs for each trainer. QUILT is implemented at the local 
school by a team of trainers who are trained by QUILT devel- 
opers. Column 3 lists the requirements and costs associated 
with individual participating teachers. (For a review of how 
the program operates, see also a previous section of this docu- 
ment, "Features: How the Program Operates.") 

Evidence for Claim 1 

A. Claim 1: Significant Increase in Teacher Knowledge and 

Understanding of Effective Classroom Questioning 

After one year's participation in the QUILT program, teach- 
ers in 13 schools showed significant gains (p=0.0001) on a 
project-developed instrument. Questionnaire on Effective Class- 
room Questioning (Cronbach's alpha = 0.76), designed to mea- 
sure knowledge, understanding, and application of selected 
concepts related to effective classroom questioning. These 
teachers also scored significantly higher than teachers in two 
alternate treatment groups. 

B. Description of Methodology for Claim 1 

1. Design. The design was a randomized, pre-post, 

comparison group design — sometimes referred to as a 

mixed between and within subjects design. A total of 

1,178 teachers from 42 schools in 13 school districts 
R | ( were the subjects of a four-state field test of the QUILT 



program: that spanned the 12-month period of May 
1991 -May 1992. District superintendents identified at 
least three schools that were interested and willing to 
take part in this field test. In 12 of the districts, faculty 
participation was mandatory, and whole faculties par- 
ticipated in the assigned treatment. In Fort Knox, teach- 
ers from three schools volunteered to participate in the 
treatment of their choice. 

The between-subjects' independent variable was 
treatment condition (i.e., the type of staff develop- 
ment provided). Under this particular design, sub- 
jects involved in the QUILT program constituted the 
experimental group; other subjects were assigned to 
one of two more traditional staff development experi- 
ences (treatmentgroups). Because of limited resources, 
the field test did not include a no treatment group. 
Highlights of the three treatments follow. 

QUILT (Condition A): Teachers participated in all 
four components of the QUILT staff develop- 
ment program over an entire school year — in- 
cluding a three-day (18-hour), preschool induc- 
tion training; seven collegiums; seven 
practicums — seven opportunities to observe, be 
observed, and engage in feedback conferences 
with one's partner; and individual study and 
classroom applications. 

Induction only (Condition B): Teachers partici- 
pated. in one component of the QUILT pro- 
gram — the three-day induction training. No 
additional training or support was provided. 

Awareness (Condition C): Teachers attended a 
three-hour inservice session on the topics con- 
tained in the QUILT Model for classroom ques- 
tioning. Of necessity, thecontent was condensed 
and presented in a more didactic manner than in 
the three-day induction training. 

The program evaluator randomly assigned one 
school from each district to each of these three treat- 
ments. The evaluator had no information concerning 
the schools other than the school names. One conse- 
quence of the random assignment of schools was a 
slightly unequal group size across the three condi- 
tions. Three hundred and ninety-seven subjects taught 
in the Condition A schools; 326 in the Condition B 
schools; and 455 in Condition C schools. 

QUILT codevelopers designed the materials and 
workshops for all three treatments. Local facilitator 
teams — trained by QUILT codevelopers in a six-day 
training-for-trainers — delivered the three-day induc- 
tion training to Condition A and Condition B teachers 
using the QUILT Induction Manual and complemen- 
tary materials. These same facilitator teams delivered 
the awareness session to Condition C teachers using 
special manuals prepared for this purpose. Post- 
workshop evaluations indicate that the goal of pro- 
viding quality experiences to each of the three groups 
was achieved. See Attachment 3 for a complete listing 
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Table 1 

QUILT Implementation Costs and Requirements, 1994-95 



Requirements for School 



Requirements for 
School Trainers 



Requirements for 
Teachers 



January 1994 

Provide information about 
QUILT to school faculties. Be- 
cause the QUILT program re- 
quires a large commitment of 
time and energy, districts 
should work collaboratively 
with schools to decide about 
the implementation of QUILT. 

February 1, 1994 
Completed applications are due 
to AEL. 

April 1994 

Select school training team (two 
to five members). Support 
preparations for QUILT at the 
school. 

June 1994 

Sponsor the school trainers to 
the 6-day training-for-trainers 
in Lexington, KY. 

August 1994 

Sponsor the induction work- 
shop for school faculty. Pro- 
vide support to the school train- 
ers for a successful induction. 

September 1994-May 1995 
Provide support to the school 
training team for implement- 
ing collegiums and partnering 
activities throughout the year. 

November 1994 
Sponsor travel costs for the 
school training team to attend a 
2-day booster. 

Cost: $250 per school for 
materials. 

Includes eight videotapes for 
trainers, complete set of more 
than 100 training transparen- 
cies, and camera-ready copy for 
QUILT poster quotes. (See last 
column for optional incentives 
to provide for teachers and 
trainers.) 



Local teacher trainers — not outside "ex- 
perts"— facilitate the QUILT training 
and implementation. QUILT staff train 
these local trainers at two events. Each 
school training team should consist of 
two to five trainers, depending on the 
size of the school and the number of 
teachers who plan to participate. 

June 19-24, 1994 — Lexington, KY 
Participate in an intensive, 6-day train- 
ing-for-trainers, which incorporates the 
QUILT induction and an overview of 
the entire QUILT program. 



August 1994 
Participate in the inten- 
sive, 3-day QUILT Induc- 
tion Workshop led by 
school trainers. This 
highly interactive train- 
ing forms the knowledge 
base for successful imple- 
mentation of QUILT. 

September 1994-May 
1995 

Implement QUILT in the 
classroom. 



August 1994 

Plan and conduct the 3-day QUILT In- 
duction Workshop for their school fac- 
ulty. 

September 1994-May 1995 
Lead the seven collegiums, which are 
90-minute meetings during which teach- 
ers review one critical component of 
QUILT, and share successes and prob- 
lems in implementing QUILT with stu- 
dents. 

Facilitate teacher efforts to partner and 
implement in the classroom. 

November 1994 — Lexington, KY 
Attend a 2-day booster. Focus will be 
on school trainers' role as facilitators of 
the change process within the school. 

Cost: $625 per trainer plus travel to 
training events, lodging, and addi- 
tional meals. 

The $625 includes registration fees for 
two training events (eight days), a com- 
plete set of trainer materials, refresh- 
ments, and lunches. Each trainer re- 
ceives the following materials: QUILT 
Induction Manual , Induction Lecturettes, 
Collegiums and Practicums: A Facilitator's 
Guide , QUILT Readings, seven collegium 
booklets, PDK Fastback by J. T. Dillon, a 
QUILT logo made from felt pieces, and 
seven teacher aid ("Q") cards, as well as 
agendas and other support materials 
necessary for conducting QUILT induc- 
tion training. 



Attend seven 90-minute 
collegiums. In each 
collegium, review one 
critical component of 
QUILT, share successes, 
discuss problems, and re- 
new commitment to 
QUILT processes. 

Observe a QUILT part- 
ner at least six times dur- 
ing the year. 

Be observed by a QUILT 
partner at least six times 
during the year. 

Meet with your QUILT 
partner after every obser- 
vation to give or receive 
feedback. 

Review QUILT materials 
as needed through indi- 
vidual study and read- 
ing. 

Cost: $25 per teacher 
for complete materi- 
als. 

Includes an Induction 
Manual , seven collegium 
booklets, PDK Fastback 
by J. T. Dillon, a QUILT 
logo made from felt piec- 
es, and seven teacher aid 
("Q") cards. 



Optional Incentives 



QUILT is a comprehensive 
staff development program. 
The basic costs are outlined 
in the first three columns. 
However, some districts and 
schools choose to provide in- 
centives to teachers and train- 
ers. Examples of incentives 
that have been provided by 
QUILT districts include the 
following: 

Stipends to school train- 
ers 

— for attending the two train- 
ing-for-trainers events, 

— for planning and conduct- 
ing the 18-hour induction 
workshop, and 
— for planning and facilitat- 
ing the yearlong QUILT 
staff development pro- 
gram. 

Stipends to teachers — for 
participating in the yearlong 
QUILT staff development 
program. 

Course credit — from a col- 
lege or university for partici- 
pating teachers. 

Inservice credit — to teach- 
ers and trainers for participa- 
tion. 

Meals and refresh- 
ments — for faculty during 
the 3-day induction training 
and the seven after-school 
collegiums. 

Early release — for teachers 
to attend the seven 90-minute 
collegiums. 

Substitute teachers — so 
that teachers can observe 
partners six times during the 
year on released time. 

Miscellaneous sup- 
plies — for the training. 

Off-campus training fa- 
cility — particularly for the 
3-day Induction Workshop. 
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of participating districts and schools and Attachment 
4 for a summary-of workshop evaluation data for the 
three conditions. 

The within-subjects' independent variable was 
time between administration of the pretests and the 
posttests. The pretests were administered prior to 
program implementation (Spring 1991); posttests were 
given at the end of the year (Spring 1992). The same 
battery of pre- and posttests was administered to 
teachers in all three treatment groups at the same 
time. 

This particular design helped address the three 
major questions driving this research study: (l)How 
effective is the QUILT staff development program? 
(2) Does participation in the QUILT program have a 
positive impact on classroom questioning? (3) Com- 
pared with two more traditional kinds of inservice 
programs, is the complete, yearlong QUILT staff de- 
velopment program more effective? The pre-post 
design was appropriate for addressing the first two of 
these questions; the comparison group design was 
needed to address the third question. 

2. Sample for Claim 1. The Questionnaire on Effec- 
tive Classroom Questioning (QECQ) was adminis- 
tered to the total population of 1,178 (K-12) teachers in 
42 schools. (See "Settings and Participants" for a 
complete description of participants.) 

3. Instruments and Procedures Used for Claim 1. 
The QECQ measures teacher knowledge about and 
understanding of classroom questioning and its rela- 
tionship to student learning and thinking. Project 
staff developed this instrument after an extensive 
search failed to turn up an existing instrument. Crite- 
ria for acceptance of this instrument included: corre- 
spondence between content of test items and content 
of QUILT; a sufficient degree of difficulty to yield a 
reasonable level of score variability needed to assess 
pre- to posttest change; and an acceptable level of 
reliability. 

This 49-item multiple choice instrument does have 
high content validity. The first version of the QECQ 
had 30 items; internal consistency reliability was 0.63. 
The developers revised problematic items and added 
new items to address additional content areas. In its 
final form, Cronbach's alpha = 0.66 on pretest; alpha 
= 0.76 at posttest administration when teachers are 
more knowledgeable about the content of the instru- 
ment. 

The 49 items are distributed on six subscales: 
effective questioning (general concepts), 6 items; 
teacher feedback and reaction, 15 items; discussion vs. 
recitation, 5 items; respondent selection and response 
formats, 7 items; cognitive levels, 9 items; and wait 
times, 7 items. 

4. Data Collection. Each district named a staff 
person to coordinate local data collection activities. In 
March 1991 and again in March 1992, these 13 persons 
were trained by QUILT staff. To assure consistency 



across sites, AEL produced an instructional videotape 
to be shown at the time instruments were adminis- 
tered. 

Teachers in each of the 42 schools met as whole 
groups to complete the QECQ following instructions 
given on the videotape. Each teacher used a special 
six-digit code to assure anonymity. The codes were 
entered on a scantron along with answers to the 49- 
item QECQ. Coordinators were instructed to collect 
all instruments and scantrons and place them imme- 
diately into envelopes, which were sealed in the teach- 
ers' presence and mailed. At AEL, the scantrons were 
examined, cleaned up as needed (stray marks erased 
and answers clearly recorded), scanned, put onto 
disk, and mailed to the project evaluator. The evalu- 
ator checked each record for unusual results; AEL 
staff matched data against original scantrons for all 
questionable instruments. 

5. Data Analysis. The following analyses were 
conducted: 

a. Univariate summary statistics were computed 
for pretest results, posttest results, and pre- posttest 
results. Included were tests for normality and 
provision of data for computation of Fmax statis- 
tics for checking analysis of variance assump- 
tions. These results were used to compute effect 
sizes. The pretest standard deviation for partici- 
pant scores in all three groups was used as the 
base for the effect size. The posttest minus pretest 
means were divided by the overall standard de- 
viation to obtain the effect size. 

b . The General Linear Model (GLM) Analysis of 
Variance was used because of unequal group 
sizes. The GLM procedure was conducted as a 
mixed design, with a between subjects factor (con- 
dition) and a within subjects factor (testing time). 
Of primary concern were two planned followups 
of the interaction. Since these comparisons were 
in the planned mode, the significant interaction of 
condition and time was not required to conduct 
these followups. 

c. The first followup procedure involved the 
comparison of pre- and posttest means within 
each condition. These were compared using di- 
rectional, dependent t tests with alpha set at 0.05. 

d. The second followup procedure involved the 
comparison of posttest means of Condition A 
with each of the other groups (A with B and A 
with C). These were compared using directional, 
Dunnett t tests with alpha set at 0.05. TheDunnett 
is specifically designed to compare groups with a 
control group or the situation where all groups 
are compared with only one other group. Dunnett 
controls Type I error rate in an experiment-wise 
manner. It is one of the few planned followup 
procedures that can be used to test directional 
hypotheses. Thus, it has high statistical power. 



Note: Table 2 and Figure 3 include only teachers for whom AEL received completed, usable-pre- and posttest QECQ. 
Because it was the end of the school year, some failed to administer the posttest. Others were lost because -of incorrect 
codes or loss/ transfer of teachers over 12 months. 

Table 2 

Pre- and Posttest Comparisons on Questionnaire on Effective 
Classroom Questioning (QECQ), Percent Correct 

Alternative Treatments 

Condition A Condition B Condition C. 

xj _ 297 N = 200 N = 292 Group Differences in Means* 
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1.17 






0.64 






0.24 


*p<0.05 





but is limited to the number of groups, minus one, 
pairwise comparisons. 

e. The third followup procedure involved the 
comparison of the pre- to posttest change mean of 
Condition A with each of the other groups (A with 
B and A with C). These were compared using 
directional, Dunnett t tests with alpha set at 0.05. 

C Description of Results for Claim 1 

Teachers who participated in the QUILT program (Condi- 
tion A) significantly increased their knowledge and under- 
standing of the research base undergirding effective classroom 
questioning. In fact, even those teachers receiving content via 
alternate treatments (Conditions B and C) showed a significant 
increase in their composite scores — confirming the content 



validity of the QECQ and the effectiveness of all three work- 
shop designs. Table 2 shows the pre- and posttest mean scores 
for all three conditions, illustrating that each condition had 
significant within group pre- to posttest differences. 

The pre- to posttest change for Condition A teachers was, 
however, much greater than those for the other two conditions 
as evidenced by the effect sizes: for Condition A, the effect size 
was +1.17, clearly higher than those for Condition B (+0.64) 
and Condition C (+0.25). 

QUILT (Condition A) teachers' pre- and posttest scores for 
the six subscales of the QECQ are illustrated in Figure 1. This 
figure clearly depicts the significant pre- to posttest mean 
changes within Condition A on all six subscales and the total 
score. The complete statistical table of pre- and posttest scores 
for the six subscales for all three treatment groups appears as 
Attachment 5. Effect sizes range from 0.36 to 1.30. 
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Figure 1. Pre- and Posttest Comparisons of Condition A Teachers' Performance on QECQ, 

Percent Correct by Subscale 
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When comparing Condition - A posttest means with the 
posttest means of_the other two groups, Condition A had 
significantly higher means than both Conditions B and C on 
the total score. Condition A teachers scored significantly higher 
than Condition B teachers on three of the subscales and signifi- 
cantly higher than Condition C teachers’ on all six subscales. 
When comparing pre- to posttest change, the change for Con- 
dition A teachers was significantly higher than the change for 
Condition B teachers on three of the subscales and on the total 
score; the Condition A teachers had a significantly higher pre- 
to posttest change than did Condition C teachers on all subscales 
as well as on the total score. 

Analysis of the QECQ data yields strong evidence in sup- 
port of Claim 1: teachers who participate in the QUILT staff 
development program significantly increase their knowledge 
and understanding of effective classroom questioning. 

D. Summary of Supplementary Evidence for Claim 1 

By self report, teachers receiving QUILT training increase 
their knowledge of effective questioning. Following the QUILT 
induction training, 340 teachers in Condition A rated personal 
knowledge gain on each of 14 specific content areas, using the 
rating scale: 1 = None to 5 = Very Much. The means for the 14 
items was 4.1. Four items had means above 4.2: wait time II 
(4.5), wait time 1 (4.3), QUILT Model (4.3), and recitation vs. 
discussion (4.2). A summary of these results appears in At- 
tachment 4. Clearly, respondents felt they had gained in 
knowledge. 

Evidence for Claims 2 and 3 

A. Claim 2: SignificantlncreaseinTeacherUseofEffective 
Questioning Practices 

A random sample of teachers from 13 schools was video- 
taped before and after participation in the QUILT program. 
Using a project-developed instrument, Classroom Question- 
ing Observation Instrument (rater reliability >0.90), trained 
coders documented teacher use of discrete questioning behav- 
iors targeted by the QUILT program. Teachers showed signifi- 
cant, positive changes in their use of the following seven 
behaviors: 

• decrease in number of teacher questions (p = .001) 

• use of wait time I (p = .008) 

• use of wait time II (p<. 001) 

• percent of questions at higher cognitive levels (p = 

.033) 

• redirection of questions (p = .009) 

• designation of respondent before question (p = .005) 

• decrease in repetition of student answers (p = .03) 

Additionally, teachers in the QUILT program performed sig- 
nificantly better than did teachers in at least one of the alternate 
treatment groups in their use of the first four behaviors listed 
above. 

Claim 3: Significant Increase in Student Thinking at 
Higher Cognitive Levels 

Student answers to teacher questions were at cognitive 
levels above simple recall significantly more often (p = .04) 
O i their teachers participated in the QUILT program. The 
>onses of these K-12 students, who were in the classrooms 



of the random sample of teachers videotaped, were coded 
using the Classroom Questioning Observation Instrument (rater 
reliability >0.90). 

B. Description of Methodology for Claims 2 and 3 

1. Design. (Same as design for Claim 1.) 

2. Sample for Claims 2 and 3. A special sample consist- 
ing of 150 teachers — 50 each from Condition A, Condition 
B, and Condition C was drawn at random. Teachers in this 
special sample were selected as subjects to be videotaped 
for 15 minutes of a lesson in which questioning was the 
intended, primary instructional method. The sample rep- 
resented approximately 12 percent of the total population, 
an intentional oversampling to compensate for the attri- 
tion that would likely occur between pre-QUILT and post- 
QUILT observations. The sample was extremely repre- 
sentative of the total population. Fifty-four percent (54%) 
of the sample (and 58 percent of the total population) 
taught at the secondary level. Seventy-five percent (75%) 
of the sample was female, compared to 77 percent of the 
total group; 96 percent were Caucasian, compared to 95 
percent of the entire population; and 14 percent were ages 
50-59, while 13 percent of the universe in this study were 
in this age bracket. The sample also matched the total 
population with regard to educational background and 
years of teaching experience, with 42 percent in the sample 
holding a bachelor's degree as their highest academic 
degree compared to 40 percent of the entire group. Sixteen 
percent (16%) of both the sample and the whole group had 
5-9 years of teaching experience. The mean number of 
years taught by teachers in the sample was 13; by the total 
population of teachers, 12. 

3. Instruments and Procedures for Claims 2 and 3. The 
instrument used in connection with Claims 2 and 3 is the 
Classroom Questioning Observation Instrument (CQOI), 
which assesses teacher and student use of effective ques- 
tioning practices from videotaped classroom excerpts. 
Developed by an external consultant who was trained 
and experienced in classroom observation and data col- 
lection, the instrument has high content validity. The 
CQOI is a low-inference, multiple-code category-system 
observation instrument. The behaviors coded exactly 
matched the observable behaviors taught in the QUILT 
program; definitions for coded behaviors were reviewed 
and validated by QUILT developers. Four middle and 
high school teachers were selected and trained as coders. 
Trainingemphasized accuracy in coding classroom ques- 
tioning behaviors and responses; speed was not an area 
of concern since coders could replay videotapes. Coders' 
reliability following the 15-hour training and indepen- 
dent work sessions ranged from 90 percent to 94 percent, 
at least matching the previously determined level of 
acceptability (.90). Individual coder reliability was de- 
termined by comparing their coding with a criterion 
videotape previously coded by the CQOI developer- 
coder trainer. 

Potential for coder bias was controlled; coders were 
not trained in the QUILT program, never met the QUILT 
program developers, were unfamiliar with the research 
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design, and were blind to the treatment conditions of any 
videotaped teacher. 

4. Data Collection for Claims 2 and 3. The names of 
teachers randomly selected by AEL staff to be part of the 
special sample for videotaping were given to district 
coordinators. The local coordinator contacted individu- 
als and asked them to participate by being videotaped 
for a 15-minute segment in which questioning was the 
primary method of instruction. Videotaping for the 
pretest took place in April and May 1991; post-treatment 
videotapes were filmed in April and May 1992. The 
number of teachers for whom both pre-QUILT and post- 
QUILT videotapes were obtained was 37 (A), 28 (B), and 
39 (C). Teachers received written instructions about the 
videotape, including a rationale for the research effort. A 
copy of "Teacher's Instructions" is included in Attach- 
ment 6. 

5. Data Analysis. The same statistical procedures were 
used to analyze these data as were used in the analysis of 
data for Claim 1. 

C. Description of Results for Claim 2. 

Seven teacher variables are reported from the CQOI. These 
data, summarized in Table 3, clearly demonstrate an increase 
in teacher use of effective questioning practices. For Condition 
A, pre- to posttest changes for all of the variables had effect 
sizes of 0.4 or greater. There were statistically significant pre- 
to post-differences at p <0.05 on all variables for Condition A. 
For both Conditions B and C, there was only one variable with 
an effect size greater than 0.4. In Condition B, teachers made 
only one significant change pre- to posttest; whereas, teachers 
in C made two significant changes. 



Teachers showed significant, positive changes in their use?; ; 
of seven behaviors. The reader is referred to an earlier section, 
"Background, Foundation, and Theoretical Framework," in 
this document for brief summaries of research about each of 
the following behaviors. 

1. Number of Questions. Condition A teachers decreased 
the number of questions asked from a mean of 41.4 to 31.0 
(p<0.001), with an effect size of -0.65. 

2. Wait Time I. Teachers significantly increased their use 
of wait time I — pausing for at least three seconds after the 
asking of a question. Before QUILT, Condition A teachers 
waited the prescribed three seconds following only 12.8 
percent of the questions posed; after QUILT, they paused 
at least three seconds following 25 percent of questions 
asked — almost doubling their use of wait time I. The effect 
size for the pre- to posttest change in wait time I was 0.99. 
Not only did Condition A teachers increase significantly 
in meeting the wait time I criterion, they made significant 
movement toward meeting the wait time I criterion as 
illustrated in Figure 2. As shown in Figure 2, teachers had 
more than a 20 percent decrease in the percent of questions 
after which they had no pause. Additionally, this figure 
graphically contrasts Condition A teachers' gains with 
those of the other two treatment groups. Both Table 3 and 
Figure 2 provide convincing evidence of the effectiveness 
of the Condition A treatment when compared to the two 
alternate treatments. (See Attachment 7 for complete sta- 
tistical data underpinning Figure 2.) 

3. Wait Time II. Even more impressive gains were real- 
ized in QUILT teachers' use of wait time II where the effect 
size was 1.72. Notice that teachers in the other two treat- 
ment groups made no significant gains in this area. Again, 
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Figure 2. Wait Time I: Percentage of Questions After Which Teachers Pause for 
Times Ranging from a Half Second to at Least Three Seconds 
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Table 3 

Pre- and Posttest Comparisons of Observational Data 
1991-92 Field Test of QUILT 
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3+ seconds 



the graphic display (Figure 
3) of the more discrete 
changes in wait time II shows 
additional development in 
the desired direction, with a 
very substantial and highly 
significant decrease in the no 
pausing behavior following 
a student response. (See At- 
tachment 8 for data related 
to Figure 3.) 



no pause Ittt ll W l ti 1 second 



ilHIHHUl .5 seconds 2 seconds 

Pretest, All 3 Conditions 





92.8% 










1.6% 




0.4% 






1.4% 


0.8% 





Posttest, Condition C 







j;7.3% 






1 2.6% | |l.5%||: 


1.0% | 



Posttest, Condition B 





! 94 % i 


1 




Posttest, Condition A 


1 2.3% 


1 |2.3%l|o.6%| 


||:|||||| |||i| llllll 64.6% !!||f || || i -i 


iii:| ; 17.5% 




i 


\ 



NOTE: May not equal 100% because all percentages 
are rounded to the nearest .1 percentage. 



Figure 3. Wait Time II: Percentage of Questions After 
Which Teachers Pause Following Student Response 



4. Cognitive level of ques- 
tion — percent of questions 
above recall. Every ques- 
tion posed on the videotapes 
was coded into one of three 
cognitive levels, using a sim- 
plified version of the Bloom 
taxonomy: Recall, Use (Un- 
derstanding, Application, 
or Analysis), or Create (Syn- 
thesis or Evaluation). 

QUILT teachers showed a 
significant increase in the 
percent of questions posed 
at cognitive levels above 
recall, with an effect size of 
0.43. Again, the other two treatment groups showed no 
significant difference pre- to posttest. The increase for 
Condition A teachers was in their questioning at the 
highest cognitive level, create (synthesis and evaluation 
on the Bloom Taxonomy). Certainly there is a linkage 
between teacher improvement in higher level question- 
ing and student responding at higher levels of thinking 
(Claim 3 below). 

5. Question redirected to other students. The percent- 
age of time that Condition A teachers asked one question 
of more than one student increased almost 10 percent 
(p<0.01) following a year in QUILT — with an effect size 
of 0.59. There were no significant changes pre- to posttest 
for the other two conditions; in fact, there was an actual 
decrease in the percentage of times that these teachers 
used this questioning strategy. 



6. Studentdesignatedafterquestion. Prior to the QUILT 
program, teachers in Condition A followed this proce- 
dure 84.1 percent of the time, leaving relatively little 
room for improvement. All the more impressive, then, is 
the almost 7 percent gain to 90.8 percent, (p<0.01) and the 
effect size of 0.59. 

7. Teacher repeats student answer. This frequently 
used and difficult to change behavior did not, in fact, 
significantly change for teachers in Conditions B and C. 
However, teachers who participated in the QUILT pro- 
gram did significantly decrease their repetitions of stu- 
dent responses (p<0.05); the effect size was -0.43. 

C. Description of Results for Claim 3 

Table 4 summarizes the data related to the cognitive levels 

of student responses. (Student responses were coded into one 



Table4 

Pre- and Posttest Comparisons of Observational Data, 1991-92 QUILT Field Test. 
Percentage of Student Responses Above the Recall Level 



Condition A 
n = 37 teachers 



Pre Post p 

28.4 37.6 <0.05 

SD 21.7 25.4 



ES 



Pre 



Condition B 
n = 28 teachers 



Post p 



Condition C 
n = 30 teachers 



ES 



Pre Post p 



0.41 



38.2 

23.3 



35.9 

28.5 



nsd 



- 0.10 



25.2 

21.8 



29.6 

20.4 



nsd 



ES 

0.19 



ERJO 



ffect Size 
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of three cognitive levels, as were teacher questions.) Almost 10 
percent more of student answers to QUILT-trained teachers 
questions were at a higher cognitive level following their teach- 
ers' yearlong participation in the QUILT program (p<.05). The 
effect size for this student change, pre- to posttest, was 0.41. 
This percentage increase closely mirrors Condition A teachers 
percentage increase in the asking of higher level questions — 
suggesting that teacher questioning affects student thinking. 
The student data from the sample in Conditions B and C 
indicated no significant difference pre- to posttest; the level of 
student responses for these groups also related closely to the 
level of teacher question. 

D. Interpretation and Discussion of Results 

1. Relationship between effect and treatment. The 
knowledge and behavior gains experienced by teachers 
who participated in the QUILT program were substantial 
and significant. Not only did these teachers significantly 
increase their knowledge and understanding of effective 
questioning practices, they translated this knowledge 
(theory) to practice. QUILT teachers showed significant 
growth and development in their use of seven identified 
questioning behaviors including rate of questioning, use 
of wait times I and II, questioning at higher cognitive 
levels, redirection of questions, designation of respondent 
after asking of question, and decrease in repetition of 
student answers. These changes in questioning proce- 
dures and techniques are attributable to teacher participa- 
tion in the seven collegiums, seven observation and feed- 
back conferences, and individual application components 
of the QUILT program. Failure of teachers in the other two 
treatment groups to make as many significant changes in 
their use of these identified behaviors — in spite of the fact 
that both groups significantly increased their knowledge 
in this area — is strong evidence to support the yearlong 
QUILT treatment. Even those teachers who participated 
in the intensive, three-day QUTLT induction did not change 
significantly in six of the seven variables. When teachers 
do not have long-term opportunities for demonstrations, 
practice, and feedback as they seek to change deeply en- 
trenched behaviors, they are unable to change these rou- 
tinized patterns. The collegiums and partnering experi- 
ences provided these opportunities. Additionally, these 
components provided reinforcement and support to these 
subjects who were attempting significant change. 

A related, and ultimately the bottom-line, goal of the 
QUILT program is to enhance student thinking. Students 
in classrooms of QUILT-trained teachers significantly in- 
creased the percent of their responses at higher levels of 
cognition pre- to posttest. This hoped-for outcome can be 
directly linked to a number of changes in teacher behavior; 
it can be indirectly linked to other features of the QUILT 
treatment. As teachers asked more questions at higher 
levels, students provided more answers at higher cogni- 
tive levels. In addition, teacher use of wait times I and II 
provided more time for student thinking prior to and 
during the answering of a question; this is also related to 
student responding at higher levels of cognition. Finally, 
O as a part of their individual study and application, QUILT 
E RlC teachers teach their students to think about their question- 



ing and answering; more specifically, they teach students 
what is called for in a response at the higher levels^ of 
cognition. 

2. Control of Rival Hypotheses. Rival hypotheses to the 
claims relate to four types of validity: (a) statistical conclu- 
sion, (b) internal, (c) construct, and (d) external. Space 
does not permit a discussion of the more than 30 recog- 
nized threats to validity; however, project evaluators made 
a conscious effort to control or minimize each through the 
evaluation design and methodological considerations. 

(a) To ensure statistical conclusion validity, data analy- 
sis assumptions were checked; planned, directional hy- 
potheses were used; methods selected had high statistical 
power; methods accounted for unequal group sizes; and 
data collection instruments and procedures had accept- 
able levels of reliability. 

(b) Most of the threats to internal validity— including 
history, maturation, testing, instrumentation, statistical re- 
gression, selection, mortality, and interactions with selec- 
tion — aj-e controlled or balanced across treatment groups 
through randomization. Schools, and thus their teachers, 
were randomly assigned to one of the three QUILT treat- 
ments, and the selection of participants for the special 
sample was based on simple random sample selection 
procedures. 

Some internal validity threats are not controlled by 
randomization. Diffusion or imitation of treatments, com- 
pensatory equalization of treatments, compensatory ri- 
valry, and resentful demoralization are problems when 
participants in different treatment groups are able to com- 
municate with each other about their treatments. While 
these cannot be totally controlled in most field situations, 
the QUILT field test minimized these threats by having 
treatments assigned on a whole school basis rather than 
having multiple treatments occurring within a school. 

Another rival hypothesis associated with this threat is 
that teachers staged their behaviors for the camera during 
the videotaping of the 15-minute questioning episodes; 
that teachers would not necessarily be using QUILT be- 
haviors "off camera." This is a potential threat any time an 
observation — "live" or videotaped — intrudes on the usual 
private teaching-learning transaction. Instructions to teach- 
ers in all three treatment groups were the same, and pre- 
test and posttest instructions were identical. 

(c) Threats to construct validity involve the possibility 
that the treatments are not well-defined, not implemented 
in ways intended (such as Rosenthal effect), or are viewed 
by participants in different ways than intended (such as 
the Hawthore effect). Great care was used in the design of 
treatments and in the delivery of training to QUILT train- 
ers to minimize construct validity threats. In addition, the 
use of a single treatment (as opposed to multiple treat- 
ments) with each given school reduced some of these 
threats. Effects were observed over a significant period of 
time (a full year) and consistent effects were observed 
across a number of different types of schools in 13 districts 
in four states. Threats to construct validity would tend to 
lead to mixed results and lack of long-term effects. Such 
was not the case in QUILT . 
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(d) External validity threats relate to the generalizability 
of the results. Had only one school district been involved 
in the field test or had a limited set of grade levels or 
content areas, this would be a concern. However, consis- 
tent results were observed across several different schools 
representing different grade levels, content areas, and types 
of schools and across four different states. The key to 
establishing external validity is demonstrating the 
replicability. Results of the QUILT field study clearly 
indicate effectiveness in several different settings. 

In summary, rival hypotheses were controlled, or ef- 
fects minimized by using powerful statistical methods 
with planned, directional hypotheses; random assignment 
of schools to treatment conditions; random selection of 
subjects for observational data collection; within-school 
singular rather than multiple treatments; well designed 
materials and high quality of training for district-based 
trainers; and replication of treatments in several settings. 

F. Educational Significance of Results 

1. Relationship of results to needs. Increased use of 
effective questioning techniques can make a tremendous 
impact on the teaching-learning process. Educational re- 
searchers consistently report that all teachers dedicate a 
large portion of their class time (40 percent, on the aver- 
age) to questioning and that, for the most part, teachers do 
not use questioning techniques correlated to positive learner 
outcomes. Prerequisite to the improvement of practice is 
knowledge of what constitutes effective practice. The QUILT 
program resulted in significant increases in teacher knowl- 
edge of effective questioning practices and procedures. 
Knowledge alone, however, does not result in improve- 
ment — a premise confirmed by the 1991-92 QUILT field 
test in which two alternate treatment groups (Conditions 
B and C) showed significant increases in knowledge of 
effective questioning practices, but little positive change in 
their use of these practices. On the other hand, QUILT 



(Condition A) teachers significantly improved theirper- 
formance in seven questioning behaviors. The QUILT 
program compels teachers to reflect upon their classroom 
practices — to think about the what, how, and why of their 
questions and interactions with students; to compare their 
personal behaviors with "best practice"; and to collaborate 
with peer partners in pursuit of personal improvement 
objectives. QUILT also promotes a schoolwide climate 
and culture that nurture individual change. 

While QUILTs primary focus is upon changing teacher 
questioning behaviors, its ultimate goal is to impact stu- 
dent learning and thinking. Students become more ac- 
countable for their own learning in QUILT classrooms 
where teachers use wait times I and H, pose questions 
before designating respondents, redirect questions, and 
do not habitually repeat student answers. Moreover, stu- 
dents learn how to think and question (metacognition) 
when teachers actively instruct them in these processes 
and procedures and model their use. QUILTs classroom 
applications provide direction and structure for student 
learning in this area. Moving students beyond short an- 
swers to lengthier and more complex responses is a goal of 
most present day reform movements. Four changes in the 
questioning behaviors of QUILT -trained teachers promote 
this goal: the reduction in the total number of questions 
asked; the increase in the number of questions posed at 
higher cognitive levels; and the use of wait times I and n. 
In QUILT classrooms, students also significantly increased 
their responses at higher cognitive levels. Increases in 
student thinking is a universal goal for education. 

2. Comparison of results to results from other pro- 
grams. We could identify no other program (i.e., a process 
for changing behaviors over the long-term) that (1) fo- 
cused upon the entirety of the questioning process (i.e., the 
content was comprehensive), and (2) had been rigorously 
evaluated and found to change significantly teacher per- 
formance in this area. 
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Attachment 1 



The QUILT Model: 

Teacher Behaviors for Effective 
Questioning 

Stage 1: Prepare the Question 

♦ Identify instructional purpose 

♦ Determine content focus 

♦ Select cognitive level 

♦ Consider wording and syntax 

Stage 2: Present the Question 

♦ Indicate response format 

♦ Ask the question 

♦ Select respondent 

Stage 3: Prompt Student Responses 

♦ Pause after asking question 

♦ Assist nonrespondent 

♦ Pause following student response 

Stage 4: Process Student Responses 

♦ Provide appropriate feedback 

♦ Expand and use correct responses 

♦ Elicit student reactions and questions 

Stage 5: Critique the Questioning Episode 

♦ Analyze the questions 

♦ Map respondent selection 

♦ Evaluate student response patterns 

♦ Examine teacher and student reactions 

ERIC 
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Attachments 2 



Annotated Bibliography: 

Resources on Effective Questioning for Teachers and Staff Developers 

Dantonio, M. (1990). How can we create thinkers? Questioning strategies 
that work for teachers. Bloomington, IN: National Education Service. 

This manual, intended for use by study groups of staff development 
leaders and teachers, is designed for ongoing, interactive inservice 
to promote use of more effective questioning strategies and techniques 
within the classroom. It is not a program, but rather gives sugges- 
tions for breaking the content into manageable pieces (over time), 
providing for use in the classroom, with a peer— observer for feedback. 

Dillon, J. T. (1988). Questioning and teaching: A manual of practice . New 

York, NY: Teachers College Press. 

This easy— to— read text presents Dillon's view of questioning and makes 
distinct the different purposes of questioning in (a) recitation and 
(b) discussion. 

Hunkins, F. P. (1989). Teaching thinking through effective questioning . 
Norwood, MA: Christopher— Gordon Publishers, Inc. 

This textbook is a comprehensive look at questions in the classroom. 
Hunkins provides examples as he presents information about levels of 
questions, types of questions (cognitive vs. affective), focus and 
syntax of questions, using questioning within the classroom, and 
student questioning. 

North Carolina Department of Public Instruction, Division of LEA Personnel 
Services. (1989). Extending thinking through effective questioning . 

Raleigh, NC: Author. 

Developed as three (3) three-hour workshops, these materials cover 
three major topics related to effective questioning: (a) importance 

of questioning, (b) levels and types of questions, and (c) strategies 
and techniques for effective questioning. The workshop materials were 
developed with appropriate activities, overhead transparencies, and 
scripts for trainers. 

Wiederhold, C. (1991). The question matrix: Cooperative learning and 

critical thinking . San Juan Capistrano, CA: Resources for Teachers, Inc . 

A collection of strategies and activities to promote shared inquiry in 
classroom instruction, 

Wilen, W. W. (Ed.) (1992). Questions, questioning techniques, and effective 

teaching . Washington, DC: National Education Association. 

An anthology of articles by noted educational researchers and 
practitioners in the areas of effective questioning. 



Attachment 3 



Participants in Field Test of QUILT 



Districts 


Schools 


Barren County (KY) Schools 


Austin Tracy School 
Barren County High School 
Hiseville School 


Fort Knox (KY) Community Schools 


Fort Knox High School 
Macdonald Middle School 
Walker Middle School 


Warren County (KY) Schools 


Greenwood High School 
Moss Middle School 
Warren East Middle School 


Giles County (TN) Schools 


Elkton School 
Minor Hill School 
Southside Elementary School 


Lawrence County (TN) Schools 


Ethridge Elementary School 

Leoma Elementary School 

South Lawrence Elementary School 


Tullahoma (TN) City Schools 


Bel Aire Elementary School 
Farrar Elementary School 
Robert E. Lee Elementary School 


Augusta County (VA) Schools 


Hugh Cassell Elementary School 
Farrar Elementary School 
Verona Elementary School 


Campbell County (VA) Schools 


Altavista High School 
Brookville High School 
William Campbell High School 
Rustburg High School 


Cabell County (WV) Schools 


Barboursville High School 
Barboursville Middle School 
Geneva Kent Elementary School 
Hite-Saunders Elementary School 


Diocese of Wheeling-Charleston (WV) 


Notre Dame High School 

St* Agnes School 

St. Joseph Grade School 


Kanawha County (WV) Schools 


Dupont Junior High School 
East Bank Junior High School 
Spring Hill Junior High School 


Marshall County (WV) Schools 


Glen Dale Elementary School 
McNinch Elementary School 
Park View Elementary School 


Upshur County (WV) Schools 


Buckhannon-Upshur High School 
Buckhannon-Upshur Inter. School 
Union Elementary School 



Attachment 4 



Summary of Workshop Evaluation Data 
Condition A 



PLEASE respond to all of the items on this form. 

For the following items, please circle the number that best indicates the 
extent to which this session: 



Responding 




Not at 


all 


Mean 


Verv much 


345 


1. 


Had clear objectives 


1 


4.57 


5 


344 


2. 


Had objectives that were met 


1 


4.45 


5 


347 


3. 


Caused me to reflect on my practices 


1 


4.65 


5 


347 


4. 


Caused me to examine some of mv attitudes 


1 


4.47 


5 


344 


5. 


Facilitated development of new skills 


1 


4.22 


5 


345 


6. 


Was relevant to my needs 


1 


4.12 


5 


342 


7. 


Was conducted in a positive climate 


1 


4.75 


5 


347 


8. 


Had activities that were well sequenced 


1 


4.53 


5 


347 


9. 


Had activities that reinforced content 


1 


4.50 


5 


346 


10. 


Included appropriate examples 


1 


4.39 


5 


346 


11. 


Was conducted by a competent trainer 


1 


4.72 


5 


345 


12. 


Was conducted in a professional manner 


1 


4.77 


5 


342 


13. 


Had meaningful involvement of participants 


1 


4.50 


5 




14. 


Has stimulated me to want to use the 




4.37 




346 




materials and skills in my position 


1 


5 




15. 


Provided materials that will be useful 




4.23 




345 




to me in the future 


1 


5 



Following are content areas of the QUILT training that we would like for you 
to rate on two dimensions: personal knowledge you gained and understand- 

ability of the presentations and materials. 



Please use the scale: 1 = None 


or Not at 


all to 


5 


= very 


much 




Content Areas 


Knowledge Gain 




Understandability 


Purno8es of questions 


i 


4.05 


5 


i 


4.27 


5 


Content focus of questions 


i 


4.04 


5 


i 


4.21 


5 


Cognitive levels 


i 


3.96 


5 


i 


4.23 


5 


Wording and syntax 


i 


3.84 


5 


i 


4.10 


5 


Varying response formats 


i 


4.15 


5 


i 


4.29 


5 


Respondent selection 


i 


4.10 


5 


i 


4.29 


5 


Wait time I 


i 


4.34 


5 


i 


4.60 


5 


Wait time II 


i 


4.47 


5 


i 


4 . 55 


5 


Assisting nonrespondents 


i 


4.00 


5 


i 


4.22 


5 


Providing appropriate feedback 


i 


4.06 


5 


i 


4.30 


5 


Eliciting student dues t ions 


i 


3.98 


5 


i 


4.14 


5 


Recitation vs. discussion 


i 


4.22 


5 


i 


4 . 46 


5 


Critiouing vour questioning 


i 


4.00 


5 


i 


4.06 


5 


0UILT Model 


i 


4.32 


5 


i 


4.30 


5 



O 
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Summary of Workshop Evaluation Data 
Condition B 



PLEASE RESPOND TO ALL OF THE ITEMS ON THIS FORM. 

For the following items, please circle the number that best indicates the 
extent to which this session: 



# 



Responding Not at all Mean Very much 



320 


1. Had clear objectives 




1 


4.54 


5 




319 


2. Had objectives that were met 




1 


4.42 


5 




322 


3, Caused me to reflect on mv practices 




1 


4.58 


5 




320 


4. Caused me to examine some of my attitudes 




1 


4.42 


3 




321 


5. Facilitated development of new skills 




1 


4.17 


3 




321 


6. Was relevant to my needs 




1 


4.05 


5 




321 


7. Was conducted in a positive climate 




1 


4.64 


5 




320 


8. Had activities that were well sequenced 




1 


4.40 


5 




321 


9. Had activities that reinforced content 




1 


4.39 


5 




321 


10. Included appropriate examples 




1 


4.29 


5 




321 


11. Was conducted by a competent trainer 




1 


4.62 


5 




319 


12. Was conducted in a professional manner 




1 


4.67 


5 




316 


13. Had meaningful involvement of participants 


1 - 


4.50 


5 




320 


14. Has stimulated me to want to use the 
materials and skills in my position 




1 


4.26 


5 




321 


15. Provided materials that will be useful 
to me in the future 




1 


4.16 


5 






Following are content areas of the QUILT training that we would like for you 
to rate on two dimensions: personal knowledge you gained and understand- 

ability of the presentations and materials. 






Please use the scale: 1 = None or Not at all 


to 


3 ° very much 






Content Areas Knowledge Gain 


Understandability 




Purposes of questions 1 


4.05 




5 1 


4.25 


5 




Content focus of questions 1 


4.00 




5 1 


4.15 


5 




Cognitive levels 1 


3.98 




5 1 


4.20 


5 




Wording and svntax 1 


3.95 




5 1 


4.12 


5 




Varying response formats 1 


4.13 




5 1 


4.26 


5 




Respondent selection 1 


4.05 




5 1 


4.28 


5 




Wait time I 1 


4.44 




5 1 


4.66 


5 




Wait time II 1 


4.53 




5 1 


4.59 


5 




Assisting nonrespondents 1 


3.96 




5 1 


4.24 


5 




Providing appropriate feedback 1 


4.01 




5 1 


4.31 


5 




Eliciting student questions 1 


4.01 




5 1 


4.19 


5 




Recitation vs. discussion 1 


4.28 




5 1 


4.48 


3 




Critiquing vour questioning 1 


3.99 




5 1 


4.04 


5 




OUILT Model 1 


4.31 




5 1 


4.29 


5 




24 



Summary of Workshop Evaluation Data 
Condition C 



PLEASE respond to all of the items on this form. 

For the following items, please circle the number that best indicates the 
extent to which this session: 



Responding 




Not at 


all 


Mean 


Verv much 


372 


1. 


Had clear objectives 


1 


4.54 


5 


370 


2. 


Had objectives that were met 


1 


4.36 


5 


372 


3. 


Caused me to reflect on my practices 


1 


4.44 


5 


371 


4. 


Caused me to examine some of my attitudes 


1 


4.28 


5 


370 


3. 


Facilitated develooment of new skills 


1 


3.99 


5 


370 


6. 


Was relevant to my needs 


1 


4.01 


5 


370 


7. 


Was conducted in a positive climate 


1 


4.52 


5 


367 


8. 


Had activities that were well sequenced 


1 


4.39 


5 


368 


9. 


Had activities that reinforced content 


1 


4.22 


5 


371 


10. 


Included appropriate examples 


1 


4.17 


5 


372 


11. 


Was conducted by a competent trainer 


1 


4.65 


5 


370 


12. 


Was conducted in a professional manner 


1 


4.73 


5 


359 


13. 


Had meaningful involvement of participants 


1 


4.26 


5 




14. 


Has stimulated me to want to use the 








371 




materials and skills in my position 


1 


4.20 


5 




13. 


Provided materials that will be useful 








371 




to me in the future 


1 


4.03 


5 




Attachment 5 



Pre- and Posttest Comparisons on Questionnaire About Effective 
Classroom Questioning, Percent Correct by Subscale and Total 



Treatment Condition 



Group 

Differences 







A, n= 297 
Pre Post 


B, n= 200 
Pre Post 


C, n= 292 
Pre Post 


in 1 
Post 


Means 

Change 


Effective Questioning 


M 


40.5 52.9 


42.3 46.2 


40.8 41.1 


A>B 


A>B 


s Dre = 20.2 
Post-Pre M Diff. 


s 

ES 

P 


20.4 22.1 

0.61 
<0.001 


20.2 21.6 
0.19 
<0.05 


19.9 19.9 

0.01 
nsd 


A>C 


A>C 


Teacher Feedback and 
Reaction 

e pre = 13 - 1 
Post-Pre M Diff . 


M 

s 

ES 

P 


49.3 54.5 

13.6 15.0 

0.39 
<0.001 


49.1 52.5 

12.7 14.8 

0.25 
<0.01 


49.5 48.4 

13.0 13.9 

-0.08 
nsd 


A>C 


A>C 


Discussion vs. 
Recitation 

Bn-ra— 21.2 


M 

s 

ES 


37.1 46.1 

20.9 20.8 

0.43 


37.6 42.6 

22.1 21.2 
0.24 


35.8 39.1 

20.7 20.5 

0.16 


A>C 


A>C 


Pos?-Pre M Diff. 


P 


<0.001 


<0.01 


<0.05 






Respondent Selection 
Response Format 

s pre = ^ • 3 
Post-Pre M Diff. 


M 

s 

ES 

P 


44.7 50.9 

17.2 17.0 

0.36 
<0.001 


45.1 47.9 

18.1 18.7 
0.16 

<0.05 


40.7 43.1 

16.7 17.2 
0.14' 

<0.05 


A>C 


A>C 


Cognitive Levels 


M 


50.7 64.1 


52.3 57.8 


48.5 51.1 


A>B 


A>B 


8 pre = 

Post-Pre M Diff. 


S 

ES 

P 


18.6 20.1 
0.73 
<0.001 


18.2 19.5 

0.30 
<0.001 


17.7 19.3 

0.15 
<0.05 


A>C 


A>C 


Wait Time 


M 


50.5 78.8 


49.4 68.9 


46.3 56.2 


A>B 


A>B 


e pre“ 21.8 
Post-Pre M Diff. 


s 

ES 

P 


22.7 21.5 

1.30 
<0.001 


22.1 24.0 

0.89 
<0.001 


20.5 23.4 

0.46 
<0.001 


A>C 


A>C 


Total QECQ 


M 


46.8 58.2 


47.2 53.4 


45.1 47.4 


A>B 


A>B 


Svs»»a= 9.72 


s 

ES 


10.3 12.3 

1.17 


9.7 12.7 

0.64 


9.1 10.5 

0.24 


A>C 


A>C 


Post-Pre M Diff. 


P 


<0.001 


<0.001 


<0.001 







*p < 0.05 
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Attachment 6 



Information About the Videotaped Lesson 
For the QUILT Research Program 



You have been selected as part of a special 
sample of teachers involved in the QUILT Research . 
Program. As a consequence, you will be videotaped 
for a 15-minute segment of instruction. 


the QUILT observational system called Classroom 
Questioning Observation Instrument. The data will 
always be reported as group means; your individual 
score will never be reported. 


Background Information 

QUILT stands for Questioning and Understand- 
ing to Improve Learning and Thinking. QUILT is a 
professional development program for classroom 
teachers, designed to enhance classroom questioning 
techniques and thus to increase student learning 
and thinking. 

Your school is involved in implementing 
QUILT — or at least one portion of QUILT — and has 
agreed to collect data with which we can evaluate 
the effectiveness of the QUILT program. The video- 
tape will be used as one measure of program effec- 
tiveness. It is important for you to understand that 
this videotape will help us evaluate the effectiveness 
of the QUILT program; it will not be used to evalu- 
ate or assess your effectiveness as a teacher or 
classroom questioner. 

All of the QUILT videotapes will be sent to the 
Appalachia Educational Laboratory, where they will 
be coded by one of four teachers specially trained in 

V 


What We Want on the Videotape 

We would like a 15-minute segment of your 
teaching or reviewing a lesson of your choice in 
which classroom questioning is your primary in- 
structional technique. Please try to use your natural 
style of question-asking as much as possible. As you 
know, there are no absolute right and wrong ways of 
teaching; we simply want a sample of how you 
typically might use questioning in your current 
classroom. 

Arrange with your QUILT coordinator your 1 

choice of when to be videotaped. Please complete the 
Teacher Information Sheet attached and return it to 
your QUILT coordinator, to be returned to AEL with 
the 15-minute videotape of your class. 

We think that this research is important and 
will make a substantial contribution to the literature 
about teacher change through staff development. 

We anticipate that the results will be usefbl to 
people in school districts all across the country. 

Thanks for your help in making it possible. 


0 
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Attachment 7 



QUILT Observation data results - Year 1 
For teachers with data at both times. 



Wait-time I (Percentage of time) 



At zero seconds 



Pre 



Post 





n 


M 


SD 


M 


SD 


Change 


A 


37 


57.07 


18.81 


41.73 


32.76 


-15.33 


B 


28 


61.50 


14.46 


44.93 


28.73 


-16.56 


C 


30 


65.20 


17.56 


64.88 


33.68 


- 0.32 


T 


95 


60.94 


17.39 


49.98 


33.22 


-10.96 



At 0.5 seconds 



Pre 



Post 





n 


M 


SD 


M 


SD 


Change 


A 


37 


8.10 


11.01 


3.73 


5.38 


- 4.37 


B 


28 


6.27 


6.92 


7.62 


10.13 


1.35 


C 


30 


7.31 


6.54 


3.56 


5.91 


- 3.75 


T 


95 


7.31 


8.60 


4.82 


7.39 


- 2.49 



At 1 seconds 



A 

B 

C 



Pre 



Post 



n 


M 


SD 


M 


SD 


Change 


37 

28 

30 


10.31 

10.57 

8.11 


6.56 

7.96 

6.80 


13.01 

9.51 

9.71 


11.74 

8.08 

11.89 


2.70 
- 1.06 
1.60 


95 


9.69 


7.08 


10.94 


10.86 


1.25 



At 2 seconds 



Pre 



Post 





n 


M 


SD 


M 


SD 


Change 


A 


37 


10.28 


8.67 


15.37 


12.29 


5.09 


B 


28 


9.86 


9.05 


16.58 


12.61 


6.72 


C 


30 


7.65 


6.58 


10.00 


15.25 


2.35 


T 


95 


9.33 


8.19 


14.03 


13.53 


4.70 



At 3 or more seconds 

n 



Pre 



A 

B 

C 



O 

ERIC 
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30 



95 



M 



SD 



Post 



M 



SD 



12.82 

11.11 

10.11 



11.90 

10.05 

14.81 



24.99 

20.70 

11.50 



24.87 

19.49 

16.54 



11.46 



12 . 34 



19.47 



21.52 



Change 



12.17 

9.58 

1.39 



8.01 



Attachment 8 



QUILT Observation data results - Year 1 
P ot*, teachers with data at both times . 



Wait-time II (Percentage 
At zero seconds 

n 


of time) 

Pre 

M 


SD 


Post 

M SD 


Change 


A 37 


90.88 10.11 


64.61 


34.05 


-26.27 


B 28 


96.16 


3.93 


80.10 


24.63 


-16.05 


C 30 


92.04 


8.40 


84.67 


23.85 


- 7.37 




— — - — — - 


-- 


T 95 


92.80 


8.39 


75.51 


29.56 


-17.29 


At 0.5 seconds 


Pre 




Post 




n 


M 


SD 


M 


SD 


Change 


A 37 


1.24 


2.77 


5.61 


8.83 


4.37 


B 28 


0.95 


2.13 


2.32 


6.16 


1.36 


C 30 


2.68 


4.21 


2.61 


4.46 


- 0.07 












T 95 


1.61 


3.20 


3.69 


7.02 


2.08 


At 1 seconds 


Pre 




Post 




n 


M 


SD 


M 


SD 


Change 




_ _ _ 


— -- 




— 


— — — — — — — 


A 37 


2.38 


4.78 


17.53 


25.67 


15.15 


B 28 


1.13 


1.89 


9.39 


21.43 


8.25 


C 30 


0.56 


1.32 


7.33 


18.09 


6.77 


T 95 


1.44 


3.31 


11.91 


22.50 


10.47 


At 2 seconds 


Pre 




Post 




n 


M 


SD 


M 


SD 


Change 












A 37 


1.19 


3.10 


4.60 


8.17 


3.41 


B 28 


0.33 


1.24 


2.29 


5.73 


1.96 


C 30 


0.82 


3.15 


1.48 


3.23 


0.66 


T 95 


0.82 


2.70 


2.93 


6.33 


2.11 


At 3 or more seconds Pro 


Post 




n 


M 


SD 


M 


SD 


Change 













— — — ' 


A 37 


0.52 


1.28 


2.98 


6.73 


2.46 


B 28 


0.10 


0.51 


0.59 


1.61 


0.49 


C 30 


0.59 


2.06 


0.97 


4.57 


0.38 


T 95 


0.42 


1.43 


1.64 


5.07 


1.22 


O 












ERIC 




29 









411 P06 



OCT 02 ’96 10:54 




U.S. Department of Education 

Office of Educational Research and Improvement (OERI) 
Educational Resources information Center (ERIC) 

reproduction release 

(Specific Document) 




I. DOCUMENT IDENTIFICATION: 



Title: 



Questioning and Understanding to Improve Learning and Thinking (QUILT) 
Evaluation Results 



The 



Author(s): Appa l achiaLEducat io n al___LallQXalJ3XX J .-.Iltc 



Corporate Source: 

Appalachia Educational Laboratory, Inc. 



! Publication Date: 

i 
| 

j February 1994 

nT^E^ODUCflON RELEASE: 

In *. n»n«» lournal ol «. ERIC SSL (EDBSI « =».- ERIC »*«,. C,«. 

ss rs - - — • ■— * ^ 

„ penniscion i. »«« . - *«— « *»**' *— «• ° NE <" *» '*** *" “* ‘' 8 " 

tho bottom of th® page. 



e 1 



Check here 
For Level i Release: 

Permitting reproduction in 
microfiche (4‘ x 6* film) or 
other ERIC archival media 
(e.g.. electronic or optical) 
and paper copy. 




n 

t 



Check here 
For Level 2 Reiei 

Permitting reproductior 
microfiche (4' x 6" film 
other ERIC archival mi 
(e g., electronic or opti 
but not in paper copy. 



Level 1 



Levew-- 



Slgn 

hen-* 



Document* will be processed as indicated provided reproduction quality permits, it 
to reproduce is granted, but neither box is checked, documents will be processed at Level 1. 



thi s document as indicated above. Reproduce from the r £ non-profit 

•SZSGSSSS^^ 


Signature^^^^^ 


1 W^Z 1 


Beth D, Sattes, R S 


r D Specialist 

•wxvr- " — * 


^ar^ttoivAWrsss: 

P. 0. Box 1348 j 

Charleston, WV 25325 

* i 


Telephone: 

304/347-0414 


rXA. 

- 304/347-0487 


Address: 

i 

| sattesb@ael.org 


Date:’ 

11/4/96 




